A hierarchical indexing strategy for optimizing Apache Spark with HDFS to efficiently query big geospatial raster data     

Fei Hu  Yongyao Jiang  Yun Li  Weiwei Song  Daniel Q. Duffy 《International Journal of Digital Earth》2020,13(3):410-428

ABSTRACT

Earth observations and model simulations are generating big multidimensional array-based raster data. However, it is difficult to efficiently query these big raster data due to the inconsistency among the geospatial raster data model, distributed physical data storage model, and the data pipeline in distributed computing frameworks. To efficiently process big geospatial data, this paper proposes a three-layer hierarchical indexing strategy to optimize Apache Spark with Hadoop Distributed File System (HDFS) from the following aspects: (1) improve I/O efficiency by adopting the chunking data structure; (2) keep the workload balance and high data locality by building the global index (k-d tree); (3) enable Spark and HDFS to natively support geospatial raster data formats (e.g., HDF4, NetCDF4, GeoTiff) by building the local index (hash table); (4) index the in-memory data to further improve geospatial data queries; (5) develop a data repartition strategy to tune the query parallelism while keeping high data locality. The above strategies are implemented by developing the customized RDDs, and evaluated by comparing the performance with that of Spark SQL and SciSpark. The proposed indexing strategy can be applied to other distributed frameworks or cloud-based computing systems to natively support big geospatial data query with high efficiency.  相似文献   

南海成因及其演化模式探讨          下载免费PDF全文

李学杰  王哲  姚永坚  高红芳  祝嵩  徐子英 《中国地质》2020,47(5):1310-1322

尽管南海已进行深入的调查与研究，提出多种成因模型，包括挤出模型、弧后扩张模型、古南海俯冲拖曳模型等，但因其所处构造位置特殊，周边构造环境经历了复杂的改造，所有成因模式均未能得到广泛的认可。本文从三大板块相互作用入手，结合南海实测数据，提出南海形成的弧后扩张—左旋剪切模型。认为南海是古南海往北俯冲的弧后盆地，菲律宾海板块往北漂移形成的大规模左旋走滑是南海扩张的触发因素。印度—欧亚碰撞产生中南半岛挤出主要影响西南海盆扩张方向，使得扩张轴从近东西向转为北东向。南海及邻区晚中生代以来的演化可以分为以下阶段：1）早白垩世开始澳大利亚板块往北漂移，新特提斯洋往北俯冲消亡，导致弧后扩张，形成古南海；2）晚白垩世末—始新世，古南海往北俯冲，导致弧后拉张形成陆缘裂谷；3）早渐新世，受菲律宾海板块西缘大型左旋走滑影响，在原有裂谷的基础上从东往西海底扩张，形成南海；4）渐新世末，受俯冲后撤的影响，扩张中心往南跃迁，同时受西缘断裂左旋活动的影响，扩张轴从近东西西逐步转为北东向；5）早中新世晚期，南沙地块—北巴拉望地块与卡加延脊碰撞，南海扩张停止。  相似文献   

Bibliometric Analysis of Research Progress on Coastal Surface Water and Groundwater Interaction     

Tianzheng Cao  Dongmei Han  Xianfang Song  Wei Liu  Di Du 《地球科学进展》2020,35(2):154-166

The interactions among surface water, groundwater and seawater are closely related in the coastal area with complex hydrological conditions. A series of impacts from human activities and climate change are also more significant in this region. In order to understand the key knowledge and research status of surface water and groundwater interaction in coastal area, it is a useful method to analyze literatures in this research scope in the core database of Web of Science by using CiteSpace. The research achievements in this field were systematically sorted and potential research hotspots were explored, which may provide references for subsequent researches. The results show the following. \mathrm{①} The number of highly cited articles and highly burst articles in this research field has increased significantly since 2010. At present, this field is still in the development stage and has a broad research prospect. \mathrm{②} The United States, Australia, China and Germany have done plenty of researches on this issue and achieved a lot. At present, the number of research achievements supported by National Natural Science Foundation of China is in the lead over the world. \mathrm{③} Seawater intrusion, submarine groundwater discharge, the relationship between tide and hydrological conditions are the main research direction in this field. Hydrochemistry and isotopic analysis, and numerical simulation are the most important research methods in this field. \mathrm{④} The potential development directions and breakthroughs in this field include submarine groundwater discharge, the evolution of coastal mangrove wetlands, the migration and transformation of nutrients, the influences of different hydrological factors on coastal areas, and the impact of climate change on coastal areas. Overall, the future development of surface water and groundwater research in coastal areas is inseparable from the cross-integration of various disciplines, mutual verification of multiple methods and the introduction of new technical means.  相似文献   

Detecting arbitrarily shaped clusters in origin-destination flows using ant colony optimization     

Ci Song  Ting Ma  Yunyan Du  Hua Shu  Sihui Guo 《International journal of geographical information science》2019,33(1):134-154

An origin-destination (OD) flow can be defined as the movement of objects between two locations. These movements must be determined for a range of purposes, and strong interactions can be visually represented via clustering of OD flows. Identification of such clusters may be useful in urban planning, traffic planning and logistics management research. However, few methods can identify arbitrarily shaped flow clusters. Here, we present a spatial scan statistical approach based on ant colony optimization (ACO) for detecting arbitrarily shaped clusters of OD flows (AntScan_flow). In this study, an OD flow cluster is defined as a regional pair with significant log likelihood ratio (LLR), and the ACO is employed to detect the clusters with maximum LLRs in the search space. Simulation experiments based on AntScan_flow and SaTScan_flow show that AntScan_flow yields better performance based on accuracy but requires a large computational demand. Finally, a case study of the morning commuting flows of Beijing residents was conducted. The AntScan_flow results show that the regions associated with moderate- and long-distance commuting OD flow clusters are highly consistent with subway lines and highways in the city. Additionally, the regions of short-distance commuting OD flow clusters are more likely to exhibit ‘residential-area to work-area’ patterns.  相似文献   

琼东南盆地深水区松南-宝岛凹陷反转构造带发育特征及油气地质意义   总被引：1，自引：1，他引：0  

杨金海  杨希冰  周杰  甘军  宋爱学  江凡  杨璐 《海洋学报》2019,41(5):97-106

利用新三维地震资料对松南-宝岛凹陷反转构造带发育特征、形成期次进行研究，并从构造背景及力学机制两个方面探讨反转构造的成因机制。研究结果表明，松南-宝岛凹陷反转构造带主要发育一系列"上凸下凹"的大型褶皱背斜，伴生NWW向弱走滑断裂构造和NNW向张剪构造。反转构造及其伴生构造符合NEE右旋剪切应力场特征，形成时间与东沙运动一致，表明该反转构造带可能受晚中新世南海东北部东沙运动产生的右旋走滑应力场作用控制。反转构造有利于研究区圈闭的重建和改造，对琼东南盆地东部新区油气运移和重新优选分配的认识，具有重要的油气地质意义。  相似文献   

Assessing the impacts of ecological-living-productive land changes on eco-environmental quality in Xining City on Qinghai-Tibet Plateau,China     

ZiYi Gao  HaiFeng Zhang  XiaoNan Yang  ZhiYuan Song 《寒旱区科学》2019,(3)

The Ecological-living-productive land(ELPL) classification system was proposed in an effort to steer China's land pattern to an ecological-centered path, with the development model shifting from a single function into more integrated multifunction land use. The focus is coordinating the man-land contradictions and developing an intensive, efficient and sustainable land use policy in an increasingly tense relationship between humans and nature. Driven by socioeconomic change and rapid population growth, many cities are undergoing urban sprawl, which involves the consumption of cropland and ecological land and threatens the ecological balance. This paper aims to quantitatively analyze the critical effects of ELPL changes on eco-environmental quality according to land use classification based on leading function of ecology, living and production from 1990 to 2015 with a case study of Xining City. Also, four future land use scenarios were simulated for2030 using the Future Land Use Simulation(FLUS) model that couples human and natural effects. Our results show a decrease in productive land(PL) and an increase in ecological land(EL) and living land(LL) in Xining City. Forestry ecological land(FEL) covered the top largest proportion; agriculture productive land(APL) showed the greatest reduction and urban and rural living land(U-RLL) presented a dramatic increase. The eco-environmental quality improved in 1990-2010, mainly affected by the conversion of APL to FEL and GEL. However, the encroachment of U-RLL into APL, other ecological land(OEL) and FEL was the main contributor to the decline in eco-environmental quality in 2010-2015 as well as the primary reason for the increase area of lower-quality. The Harmonious Development(HD)-Scenario, characterized by a rational allocation of LL and PL and a better eco-environment, would have implications for planning and monitoring future management of ELPL, and may represent a valuable reference for local policy-makers.  相似文献   

Understanding geographical patterns of a city’s diurnal rhythm from aggregate data of location‐aware services     

Ting Ma  Tao Pei  Ci Song  Yaxi Liu  Yunyan Du  Xiaohan Liao 《Transactions in GIS》2019,23(1):104-117

Do collective behaviors of the daily routine of a city's inhabitants form the periodical cycling of human activity at the city level (here termed the “city's diurnal rhythm”)? If the answer is yes, do there exist geographical patterns in the city's diurnal rhythm? Using a nationwide dataset of observed uses of location‐aware services in the largest Chinese social media platform, we first confirm the significant periodicity in city‐level human activity from the perspective of the aggregate degree of social media uses over a day. We then investigate geographical changes in the diurnal rhythm of human activity and its local variations in different parts of the city, and between weekdays and weekend days, over 340 Chinese cities. Our results show that a city's diurnal rhythm across the whole country exhibits both regular, nationally conspicuous shifts along geographical gradients and locally distinct spatiotemporal changes within the city. Our findings could provide insights into the characterization of the daily routine of city‐level human activity and its geographical patterns, and have potential for several issues in terms of planning, management, and decision‐making related to human population dynamics.  相似文献   

高烈度区双洞隧道洞口段地震响应分析与振动台模型试验          下载免费PDF全文

皇民  苑俊杰  赵玉如  毕洁同  王松 《中国地质灾害与防治学报》2020,31(1):113-119

强震作用下,隧道地下结构损伤严重,其中洞口段更是抗震的薄弱环节,但其影响规律和特点目前尚缺乏系统的计算分析和研究。以强震区扯羊隧道为例,首先采用FLAC3D对双洞隧道洞口段地震响应进行分析,再用模型试验的方法进行洞口段模型试验,测量其地震响应应变规律,观测其震害发展情况,并与数值分析结果相互验证。结果表明:地震作用下,隧道仰拱横向位移较大;随着隧道埋深增加,内力逐步增加,其中墙角部位内力较大;洞口处隧道围岩在强震作用下会产生贯通性裂缝,影响隧道洞口稳定;地震作用下的明洞与暗洞交接处内力较大;地震作用下的双洞隧道之间存在较强的动力相互作用。  相似文献   

云南省新平县哀牢山地区富硒土壤成因及其影响因素研究   总被引：1，自引：0，他引：1  

黄钊  陈政  薛传东  刀艳  许胜超  代达龙  吴松 《现代地质》2020,34(3):609-617

以云南省新平县者竜—戛洒地区1∶5万土地质量地球化学调查评价结果为基础,对该地区富硒土壤的成因及其影响因素进行分析,结果表明,研究区土壤硒平均含量为0.38 mg/kg,高于全国土壤硒平均含量0.29 mg/kg,按0.4 mg/kg相似文献   

Preface to the special issue on the ecological-hydrological processes in the Heihe River Basin: Integrated research on observation,modeling and data analysis     

Song  Changqing 《地理学报(英文版)》2019,29(9):1437-1440

Journal of Geographical Sciences -  相似文献